Product Code Database
A newline (frequently called line ending, end of line ( EOL), next line ( NEL) or line break) is a control character or sequence of control characters in character encoding specifications such as ASCII, EBCDIC, Unicode, etc. This character, or a sequence of characters, is used to signify the end of a line of text and the start of a new one.
Later, in the age of modern , standardized character set control codes were developed to aid in white space text formatting. ASCII was developed simultaneously by the International Organization for Standardization (ISO) and the American Standards Association (ASA), the latter being the predecessor organization to American National Standards Institute (ANSI). During the period of 1963 to 1968, the ISO draft standards supported the use of either + or alone as a newline, while the ASA drafts supported only +.
The sequence + was commonly used on many early computer systems that had adopted Teletype machines—typically a Teletype Model 33 ASR—as a console device, because this sequence was required to position those printers at the start of a new line. The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in time to print the next character. Any character printed after a would often print as a smudge in the middle of the page while the print head was still moving the carriage back to the first position. "The solution was to make the newline two characters: to move the carriage to column one, and to move the paper up." In fact, it was often necessary to send Output padding—extraneous CRs or NULs—which are ignored but give the print head time to move to the left margin. Many early video displays also required multiple character times to Scrolling the display.
On such systems, applications had to talk directly to the Teletype machine and follow its conventions since the concept of hiding such hardware details from the application was not yet well developed. Therefore, text was routinely composed to satisfy the needs of Teletype machines. Most minicomputer systems from DEC used this convention. CP/M also used it in order to print on the same terminals that minicomputers used. From there MS-DOS (1981) adopted CP/M's + in order to be compatible, and this convention was inherited by Microsoft's later Windows operating system.
The Multics operating system began development in 1964 and used alone as its newline. Multics used a device driver to translate this character to whatever sequence a printer needed (including extra Output padding), and the single byte was more convenient for programming. What seems like a more obvious choice – – was not used, as provided the useful function of overprinting one line with another to create boldface, underscore and strikethrough effects. Perhaps more importantly, the use of alone as a line terminator had already been incorporated into drafts of the eventual ISO/IEC 646 standard. Unix followed the Multics practice, and later Unix-like systems followed Unix. This created conflicts between Windows and Unix-like , whereby files composed on one operating system could not be properly formatted or interpreted by another operating system (for example a UNIX shell script written in a Windows text editor like Notepad).
Some character sets provide a separate newline character code. EBCDIC, for example, provides an character code in addition to the and codes. Unicode, in addition to providing the ASCII and control codes, also provides a "next line" () control code, as well as control codes for "line separator" and "paragraph separator" markers. Unicode also contains printable characters for visually representing line feed ␊, carriage return ␍, and other C0 control codes (as well as a generic newline, ) in the Control Pictures block.
| +Software applications and operating system representation of a newline with one or two control characters ! scope="col" | Operating system ! scope="col" | Character encoding ! scope="col" | Abbreviation ! scope="col" | hexadecimal value ! scope="col" | decimal value ! scope="col" | Escape sequence |
In some older protocols, the new line may be followed by a checksum or parity character.
| Line Feed, |
| Vertical Tab, |
| Form Feed, |
| Carriage return, |
| () followed by () |
| Next Line, |
| Line Separator, |
| Paragraph Separator, |
While it may seem overly complicated compared to an approach such as converting all line terminators to a single character (e.g. ), because Unicode is designed to preserve all information when converting a text file from any existing encoding to Unicode and back (round-trip integrity), Unicode needs to make the same distinctions between line breaks made by other encodings. For instance EBCDIC has , , and characters, so all three have to also exist in Unicode.
Most newline characters and sequences are in ASCII's C0 controls (i.e. have Unicode code points up to ). The three newline characters outside of this range—, and —are often not recognized as newlines by software. For example:
Unicode includes some intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline:
The C language provides the \n (newline) and \r (carriage return). However, these are not required to be equivalent to the ASCII and control characters. The C standard only guarantees two traits:
On Unix operating system platforms, where C originated, the native newline sequence is ASCII (), so \n was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode is a no-op, and Unix has no notion of text mode or binary mode. This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms.
The C standard library function is best avoided in binary mode because any file not written with the Unix newline convention will be misread. Also, in text mode, any file not written with the system's native newline sequence (such as a file created on a Unix system, then copied to a Windows system) will be misread as well.
Another common problem is the use of \n when communicating using an Internet protocol that mandates the use of ASCII + for ending lines. Writing \n to a text mode stream works correctly on Windows systems, but produces only on Unix, and something completely different on more exotic systems. Using \r\n in binary mode is slightly better.
Many languages, such as C++, Perl, and Haskell provide the same interpretation of \n as C. C++ has an alternative input/output (I/O) model where the manipulator can be used to output a newline (and flushes the stream buffer).
Java, PHP, and Python provide the \r\n sequence (for ASCII +). In contrast to C, these are guaranteed to represent the values and , respectively.
The Java Class Library input/output (I/O) methods do not transparently translate these into platform-dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of , , or + as a line terminator (see ). The method can be used to retrieve the underlying line separator.
Example:
String eol = System.lineSeparator();
String lineColor = "Color: Red" + eol;
Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file.
Some languages have created special variables, constants, and to facilitate newlines during program execution. In some languages such as PHP and Perl, double quotes are required to perform escape substitution for all escape sequences, including \n and \r. In PHP, to avoid portability problems, newline sequences should be issued using the PHP_EOL constant.
Example in C#:
string eol = Environment.NewLine;
string lineColor = "Color: Red" + eol;
string eol2 = "\n";
string lineColor2 = "Color: Blue" + eol2;
Text in files created with programs which are common on Unix-like or classic Mac OS, appear as a single long line on most programs common to MS-DOS and Microsoft Windows because these do not display a single or a single as a line break.
Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extra may be displayed as a second line break, as , or as at the end of each line.
Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file.
The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. For example, a compiler may fail with obscure syntax errors even though the source file looks correct when displayed on the console or in an Text editor. Modern text editors generally recognize all flavours of + newlines and allow users to convert between the different standards. are usually also capable of displaying text files and websites which use different types of newlines.
Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save—often inconsistently.
Most textual Internet protocols (including HTTP, SMTP, FTP, IRC, and many others) mandate the use of ASCII + (\r\n, ) on the protocol level, but recommend that tolerant applications recognize lone (\n, ) as well. Despite the dictated standard, many applications erroneously use the C newline escape sequence \n () instead of the correct combination of carriage return escape and newline escape sequences \r\n (+) (see section Newline in programming languages above). This accidental use of the wrong escape sequences leads to problems when trying to communicate with systems adhering to the stricter interpretation of the standards instead of the suggested tolerant interpretation. One such intolerant system is the qmail mail transfer agent that actively refuses to accept messages from systems that send bare instead of the required +.
The standard Internet Message Format for email states: "CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body". Differences between SMTP implementations in how they treat bare LF and/or bare CR characters have led to SMTP spoofing attacks referred to as "SMTP smuggling".
The File Transfer Protocol can automatically convert newlines in files being transferred between operating system with different newline representations when the transfer is done in "ASCII mode". However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively data corruption the file. FTP clients often employ some heuristics (for example, inspection of filename extensions) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.
For example, the editor Vim can make a file compatible with the Windows Notepad text editor. Within vim
:set fileformat=dos
:wq
$ tr -d '\r' < ''inputfile'' > ''outputfile''
or, if the text has only newlines, by converting all newlines to with
$ tr '\r' '\n' < ''inputfile'' > ''outputfile''
The same tasks are sometimes performed with awk, sed, or in Perl if the platform has a Perl interpreter:
$ file myfile.txt
myfile.txt: ASCII English text, with CRLF line terminators
Other tools permit the user to visualise the EOL characters:
In text intended primarily to be read by humans using software which implements the word wrap feature, a newline character typically only needs to be stored if a line break is required independent of whether the next word would fit on the same line, such as between and in vertical lists. Therefore, in the logic of word processor and most , newline is used as a paragraph break and is known as a "hard return", in contrast to "soft returns" which are dynamically created to implement word wrapping and are changeable with each display instance. In many applications a separate control character called "manual line break" exists for forcing line breaks inside a single paragraph. The glyph for the control character for a hard return is usually a pilcrow (¶), and for the manual line break is usually a carriage return arrow (↵).
([[U|Unicode]]+008D REVERSE LINE FEED, ISO/IEC 6429 8D, decimal 141) is used to move the printing position back one line (by reverse feeding the paper, or by moving a display cursor up one line) so that other characters may be printed over existing text. This may be done to make them bolder, or to add underlines, strike-throughs or other characters such as [[diacritic]]s. The reverse line feed was called a ''line starve'' a [[pun]] on ''line feed'' in the ''Hacker's Dictionary''.
Similarly, (Unicode+008B PARTIAL LINE FORWARD, decimal 139) and (Unicode+008C PARTIAL LINE BACKWARD, decimal 140) can be used to advance or reverse the text printing position by some fraction of the vertical line spacing (typically, half). These can be used in combination for subscripts (by advancing and then reversing) and superscripts (by reversing and then advancing), and may also be useful for printing diacritics.
|
|
